# KRR: compare matched AA & WA

## Preparation

### Collection of confounds and behaviors; filter unsatisfactory subjects

```matlab
ABCD_read_all_measures('/mnt/isilon/CSC2/Yeolab/Data/ABCD/process/y0/rs_GSR', [], ...
   '/mnt/isilon/CSC2/Yeolab/Data/ABCD/process/y0/orig_scripts/release2.0/lists/subjects_pass_rs.txt', ...
   '/home/jingweil/storage/MyProject/fairAI/ABCD_race/scripts/lists', '_pass_rs')
```

### Split & match

```matlab
ABCD_match_and_split(...
   '/home/jingweil/storage/MyProject/fairAI/ABCD_race/scripts/lists/phenotypes_pass_rs.txt', ...
   '/home/jingweil/storage/MyProject/fairAI/ABCD_race/scripts/lists/subjects_pass_rs_pass_pheno.txt', ...
   'race', 'site', 'family_id', ...
   '/home/jingweil/storage/MyProject/fairAI/ABCD_race/scripts/lists/confounds_list.txt', ...
   '/home/jingweil/storage/MyProject/fairAI/ABCD_race/scripts/lists/behavior_list.txt', 100, 2.45, ...
   '/home/jingweil/storage/MyProject/fairAI/ABCD_race/mat/matchANDsplit/20200719', ...
   '_pass_rs_pass_pheno')
```

### Compare the behavioral variances between matched AA & WA

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_pheno_var_AAvsWA_matched(...
   fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719', 'sel_AAWA_pass_rs_pass_pheno.mat'), ...
   fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
   fullfile(proj_dir, 'scripts', 'lists', 'phenotypes_pass_rs.txt'), ...
   fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719', 'AAvsWA_var_Levene.mat'))
```

## ------ KRR: regress age, sex, FD, DVARS, ICV, parental education from both behaviors and RSFC ------

### Run KRR (bash)

```bash
../KRR/ABCD_KRR_reg_AgeSexMtIcvPEduc_from_y_FC.sh
```

### Plot correlation and predictive COD accuracy of all behaviors on all test subjects

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_KRR_violin_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
   'corr', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
   'corr_allsubj_violin')
ABCD_KRR_violin_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
   'predictive_COD', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
   'pCOD_allsubj_violin')
```

### Permutation test of predictability

1. metric: predictive COD

   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process');\
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'predictive_COD', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y_FC.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_y_FC/logs
   jname=perm_pCOD_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 30:00:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

2. metric: Pearson's correlation
   
   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process'); \
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'corr', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y_FC.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_y_FC/logs
   jname=perm_corr_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 30:00:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

### Compute accuracy metric per race group; get predictable behaviors

Get behavioral measures with >0.15 correlation accuracy across all test subjects:

```matlab
clear
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
bhvr_ls = fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt');
colloq_ls = fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt');
[bhvr_nm, nbhvr] = CBIG_text2cell(bhvr_ls);
colloq_nm = CBIG_text2cell(colloq_ls);
model_dir = fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC');
opt = zeros(nbhvr, 1);
for b = 1:nbhvr
   load(fullfile(model_dir, bhvr_nm{b}, ['final_result_' bhvr_nm{b} '.mat']))
   opt(b) = mean(optimal_stats.corr);
end
idx = find(opt > 0.15);
bhvr_nm = bhvr_nm(idx);
colloq_nm = colloq_nm(idx);
mkdir(fullfile(model_dir, 'lists'))
CBIG_cell2text(bhvr_nm, fullfile(model_dir, 'lists', ['R_thres0.15_' num2str(length(idx)) 'behaviors.txt']))
CBIG_cell2text(colloq_nm, fullfile(model_dir, 'lists', ['R_thres0.15_' num2str(length(idx)) 'colloquial.txt']))
```

1. metric: predictive COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_pCOD_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y_FC.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'R_thres0.15_12behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'))
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_corr_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y_FC.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'R_thres0.15_12behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'))
   ```

### Permutation test of accuracy difference

1. metric: predictive COD

   Only predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'pCOD_predictable.txt'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
      [], ...
      'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat') )
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'corr_predictable.txt'), ...
      'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
      [], ...
      'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat') )
   ```

### Violin plot

#### Plot accuracy difference

1. metric: predictive COD

   Only plot predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', ...
      'pCOD_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'pCOD_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), 'pCOD')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA([], [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), 'pCOD_allbehaviors')
   ```

2. metric: Pearson's correlation
   
   Only plot predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', ...
      'corr_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'corr_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), 'corr')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA([], [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), 'corr_allbehaviors')
   ```

#### Plot "predicted score - original score"

1. Only plot behaviors with predictable COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_violin_predVStrue(...
       fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
       'predVStrue_predictableCOD', fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
       'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), [], ...
       fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'pCOD_predictable_colloquial.txt'))
   ```

2. Only plot behaviors with predictable corr

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_violin_predVStrue(...
       fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
       'predVStrue_predictableCORR', fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
       'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC_predictable.mat'), [], ...
       fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC', 'lists', 'corr_predictable_colloquial.txt'))
   ```

3. All 36 behaviors

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';

   % Are AA-WA differences in terms of "predicted - orginal score" significant?
   ABCD_PermTest_predVStrue_AAvsWA(fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
       'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), [], ...
       fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
       'sig_predVStrue_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'))

   % plot
   ABCD_KRR_violin_predVStrue(...
       fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y_FC'), ...
       'predVStrue_allbehaviors', fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
       'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
       fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
       'sig_predVStrue_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), [])
   ```

### Optimal parameters

```matlab
ABCD_KRR_collect_opt_param('/home/jingweil/storage/MyProject/fairAI/ABCD_race/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_y_FC', [])
```

### Scatter plot between difference in BBA similarity and difference in accuracy

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_scatter_BBAdiff_vs_AccDiff(fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
    'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), fullfile(proj_dir, ...
    'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
    'predictive_COD', fullfile(proj_dir, 'mat', 'interpretation', 'KRR', '20200719', ...
    'reg_AgeSexMtIcvPEduc_fr_y_FC', 'learned_BWAS.mat'), fullfile(proj_dir, 'mat', ...
    'interpretation', 'KRR', '20200719', 'reg_AgeSexMtIcvPEduc_fr_y_FC', ...
    'real_BWAS_testAAvsWA.mat'), fullfile(proj_dir, 'figures', 'interpretation', ...
    'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_ft_y_FC', 'pCOD_vs_BBA'))
```

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_scatter_BBAdiff_vs_AccDiff(fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', ...
    'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), fullfile(proj_dir, ...
    'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_FC.mat'), ...
    'corr', fullfile(proj_dir, 'mat', 'interpretation', 'KRR', '20200719', ...
    'reg_AgeSexMtIcvPEduc_fr_y_FC', 'learned_BWAS.mat'), fullfile(proj_dir, 'mat', ...
    'interpretation', 'KRR', '20200719', 'reg_AgeSexMtIcvPEduc_fr_y_FC', ...
    'real_BWAS_testAAvsWA.mat'), fullfile(proj_dir, 'figures', 'interpretation', ...
    'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_ft_y_FC', 'corr_vs_BBA'))
```

## ------ KRR: no confounds regression ------

### Run KRR (bash)

```bash
../KRR/ABCD_KRR_no_reg.sh
```

### Plot correlation and predictive COD accuracy of all behaviors on all test subjects

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_KRR_violin_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg'), ...
   'corr', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'no_reg'), ...
   'corr_allsubj')
ABCD_KRR_violin_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg'), ...
   'predictive_COD', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'no_reg'), ...
   'pCOD_allsubj')
```

### Permutation test of predictability

1. metric: predictive COD

   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process');\
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'no_reg'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'predictive_COD', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/no_reg/logs
   jname=perm_pCOD_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 30:00:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

2. metric: Pearson's correlation
   
   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process'); \
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'no_reg'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'corr', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_no_reg.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/no_reg/logs
   jname=perm_corr_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 30:00:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

### Compute accuracy metric per race group; get predictable behaviors

Get behavioral measures with >0.15 correlation accuracy across all test subjects:

```matlab
clear
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
bhvr_ls = fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt');
colloq_ls = fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt');
[bhvr_nm, nbhvr] = CBIG_text2cell(bhvr_ls);
colloq_nm = CBIG_text2cell(colloq_ls);
model_dir = fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg');
opt = zeros(nbhvr, 1);
for b = 1:nbhvr
   load(fullfile(model_dir, bhvr_nm{b}, ['final_result_' bhvr_nm{b} '.mat']))
   opt(b) = mean(optimal_stats.corr);
end
idx = find(opt > 0.15);
bhvr_nm = bhvr_nm(idx);
colloq_nm = colloq_nm(idx);
mkdir(fullfile(model_dir, 'lists'))
CBIG_cell2text(bhvr_nm, fullfile(model_dir, 'lists', ['R_thres0.15_' num2str(length(idx)) 'behaviors.txt']))
CBIG_cell2text(colloq_nm, fullfile(model_dir, 'lists', ['R_thres0.15_' num2str(length(idx)) 'colloquial.txt']))
```

1. metric: predictive COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_pCOD_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg', 'lists', 'R_thres0.15_19behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg.mat'))
   ```

2. metric: Pearson's correlation
   
   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_corr_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_no_reg.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg', 'lists', 'R_thres0.15_19behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_no_reg.mat'))
   ```

### Permutation test of accuracy difference

1. metric: predictive COD

   Only predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg', 'lists', 'pCOD_predictable.txt'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_no_reg_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg.mat'), ...
      [], ...
      'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_no_reg.mat') )
   ```

2. metric: Pearson's correlation
   
   Only predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_no_reg_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg', 'lists', 'corr_predictable.txt'), ...
      'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_no_reg_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_no_reg.mat'), ...
      [], ...
      'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_no_reg.mat') )
   ```

### Violin plot

#### Plot accuracy difference

1. metric: predictive COD

   Only plot predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg', 'lists', ...
      'pCOD_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'no_reg', 'lists', 'pCOD_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_no_reg_predictable.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'no_reg'), 'pCOD')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA([], [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_no_reg.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_no_reg.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'no_reg'), 'pCOD_allbehaviors')
   ```

2. metric: Pearson's correlation
   
   Only plot predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'no_reg', 'lists', ...
      'corr_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'no_reg', 'lists', 'corr_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_no_reg_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_no_reg_predictable.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'no_reg'), 'corr')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_violin_withnull_AAvsWA([], [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_no_reg.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_no_reg.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'no_reg'), 'corr_allbehaviors')
    ```

## ------ KRR: regress age, sex, FD, DVARS, ICV, parental education from behaviors ------

### Run KRR (bash)

```bash
../KRR/ABCD_KRR_reg_AgeSexMtIcvPEduc_from_y.sh
```

### Plot correlation and predictive COD accuracy of all behaviors on all test subjects

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_KRR_whisker_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
   'corr', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
   'corr_allsubj')
ABCD_KRR_whisker_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
   'predictive_COD', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
   'pCOD_allsubj')
```

### Permutation test of predictability

1. metric: predictive COD

   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process');\
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'predictive_COD', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_y/logs
   jname=perm_pCOD_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 30:00:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

2. metric: Pearson's correlation

   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process'); \
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'corr', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_y/logs
   jname=perm_corr_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 00:30:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

### Compute accuracy metric per race group; get predictable behaviors

Get behavioral measures with >0.15 correlation accuracy across all test subjects:

```matlab
clear
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
bhvr_ls = fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt');
colloq_ls = fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt');
[bhvr_nm, nbhvr] = CBIG_text2cell(bhvr_ls);
colloq_nm = CBIG_text2cell(colloq_ls);
model_dir = fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y');
opt = zeros(nbhvr, 1);
for b = 1:nbhvr
   load(fullfile(model_dir, ['final_result_' bhvr_nm{b} '.mat']))
   opt(b) = mean(optimal_stats.corr);
end
idx = find(opt > 0.15);
bhvr_nm = bhvr_nm(idx);
colloq_nm = colloq_nm(idx);
mkdir(fullfile(model_dir, 'lists'))
CBIG_cell2text(bhvr_nm, fullfile(model_dir, 'lists', ['R_thres0.15_' num2str(length(idx)) 'behaviors.txt']))
CBIG_cell2text(colloq_nm, fullfile(model_dir, 'lists', ['R_thres0.15_' num2str(length(idx)) 'colloquial.txt']))
```

1. metric: predictive COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_pCOD_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'R_thres0.15_12behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'))
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_corr_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_y.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'R_thres0.15_12behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'))
   ```
3. metric: MSE (all behaviors)

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_errors_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, 'MSE', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'MSE_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'))
   ```

4. metric: normalized MSE (all behaviors, normalized by variance in training set)

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_errors_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, 'MSE_norm', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'normMSE_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'))
   ```

### Permutation test of accuracy difference

1. metric: predictive COD

   Only predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'pCOD_predictable.txt'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      [], ...
      'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat') )
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'corr_predictable.txt'), ...
      'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      [], ...
      'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat') )
   ```

3. metric: MSE

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'MSE_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      'MSE', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_MSE_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat') )
   ```

### Whisker plot

1. metric: predictive COD

   Only plot predictable behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', ...
      'pCOD_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'pCOD_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), 'pCOD')

    ABCD_violin_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', ...
      'pCOD_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'pCOD_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), 'pCOD_violin')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA([], [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), 'pCOD_allbehaviors')
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y', 'lists', ...
      'corr_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_y', 'lists', 'corr_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y_predictable.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), 'corr')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      [],  [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), 'corr_allbehaviors')
   ```

3. metric: MSE

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      [],  [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'MSE_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_MSE_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_y.mat'), ...
      'MSE', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_y'), 'MSE')
   ```

## ------ KRR: regress age, sex, FD, DVARS, ICV, parental education from RSFC ------

## Regress confounds from RSFC

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_regress_cfds_from_FC(...
   fullfile(proj_dir, 'mat', 'RSFC', 'pass_rs_pass_pheno_5351.mat'), ...
   fullfile(proj_dir, 'scripts', 'lists', 'phenotypes_pass_rs.txt'), ...
   fullfile(proj_dir, 'scripts', 'lists', 'confounds_list.txt'), ...
   fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
   fullfile(proj_dir, 'mat', 'RSFC', 'pass_rs_pass_pheno_5351_reg_AgeSexMtIcvPEduc.mat'))
```

### Run KRR (bash)

```bash
../KRR/ABCD_KRR_reg_AgeSexMtIcvPEduc_from_FC.sh
```

### Plot correlation and predictive COD accuracy of all behaviors on all test subjects

```matlab
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
ABCD_KRR_whisker_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), ...
   'corr', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), ...
   'corr_allsubj')
ABCD_KRR_whisker_acc_allsub(fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), ...
   'predictive_COD', fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), ...
   'pCOD_allsubj')
```

### Permutation test of predictability 

1. metric: predictive COD

   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process');\
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'predictive_COD', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_FC.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_FC/logs
   jname=perm_pCOD_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 30:00:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

2. metric: Pearson's correlation

   ```bash
   ssh headnode
   proj_dir=/home/jingweil/storage/MyProject/fairAI/ABCD_race
   cmd="matlab -nodesktop -nodisplay -nojvm -r \" cd('$proj_dir/scripts/Unfairness_ABCD_process');\
      ABCD_addpath; ABCD_KRR_predictable_behavior(\
      fullfile('$proj_dir', 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), \
      fullfile('$proj_dir', 'mat', 'matchANDsplit', '20200719'), \
      '_pass_rs_pass_pheno', 1000, 'corr', \
      fullfile('$proj_dir', 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_FC.mat'));  exit; \" "
   work_dir=$proj_dir/models/KRR/20200721/reg_AgeSexMtIcvPEduc_fr_FC/logs
   jname=perm_corr_predictability
   $CBIG_CODE_DIR/setup/CBIG_pbsubmit -cmd "$cmd" -walltime 00:30:00 -mem 12G \
      -name $jname -joberr $work_dir/$jname.err -jobout $work_dir/$jname.out
   ```

### Compute accuracy metric per race group; get predictable behaviors

Get behavioral measures with >0.1 correlation accuracy across all test subjects:

```matlab
clear
proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
bhvr_ls = fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt');
colloq_ls = fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt');
[bhvr_nm, nbhvr] = CBIG_text2cell(bhvr_ls);
colloq_nm = CBIG_text2cell(colloq_ls);
model_dir = fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC');
opt = zeros(nbhvr, 1);
for b = 1:nbhvr
   load(fullfile(model_dir, ['final_result_' bhvr_nm{b} '.mat']))
   opt(b) = mean(optimal_stats.corr);
end
idx = find(opt > 0.1);
bhvr_nm = bhvr_nm(idx);
colloq_nm = colloq_nm(idx);
mkdir(fullfile(model_dir, 'lists'))
CBIG_cell2text(bhvr_nm, fullfile(model_dir, 'lists', ['R_thres0.1_' num2str(length(idx)) 'behaviors.txt']))
CBIG_cell2text(colloq_nm, fullfile(model_dir, 'lists', ['R_thres0.1_' num2str(length(idx)) 'colloquial.txt']))
```

1. metric: predictive COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_pCOD_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_FC.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', 'R_thres0.1_15behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'))
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_KRR_corr_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'behavior_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'colloquial_list.txt'), ...
      fullfile(proj_dir, 'scripts', 'lists', 'subjects_pass_rs_pass_pheno.txt'), ...
      fullfile(proj_dir, 'mat', 'matchANDsplit', '20200719'), ...
      '_pass_rs_pass_pheno', 120, ...
      fullfile(proj_dir, 'mat', 'predictability', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPEduc_fr_FC.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', 'R_thres0.1_15behaviors.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'))
   ```

### Permutation test of accuracy difference

1. metric: predictive COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', ...
      'pCOD_predictable.txt'), 'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'), ...
      [], 'predictive_COD', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat') )
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat'), ...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', ...
      'corr_predictable.txt'), 'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat') )
   ```

   All behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_PermTest_AAvsWA( ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'), ...
      [], 'corr', ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat') )
   ```

### Whisker plot

1. metric: predictive COD

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', ...
      'pCOD_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', 'pCOD_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), 'pCOD')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      [],  [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_pCOD_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'), ...
      'predictive_COD', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), 'pCOD_allbehaviors')
   ```

2. metric: Pearson's correlation

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      fullfile(proj_dir, 'models', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', ...
      'corr_predictable.txt'),  fullfile(proj_dir, 'models', 'KRR', '20200721', ...
      'reg_AgeSexMtIcvPEduc_fr_FC', 'lists', 'corr_predictable_colloquial.txt'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC_predictable.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), 'corr')
   ```

   Plot all behaviors:

   ```matlab
   proj_dir = '/home/jingweil/storage/MyProject/fairAI/ABCD_race';
   ABCD_whisker_AAvsWA(...
      [],  [], ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'), ...
      fullfile(proj_dir, 'mat', 'AAvsWA', 'KRR', 'sig_corr_pass_rs_pass_pheno_reg_AgeSexMtIcvPeduc_fr_FC.mat'), ...
      'corr', ...
      fullfile(proj_dir, 'figures', 'AAvsWA', 'KRR', '20200721', 'reg_AgeSexMtIcvPEduc_fr_FC'), 'corr_allbehaviors')
   ```
